This invention relates to a neural network circuit for executing image recognition processing or the like.
Much interest has recently been shown in a field of neutral network in data processing. The neural network is brought up from simulation of a neuron structure of a brain of a living thing. Many neural networks are accomplished by a conventional yon Neumann sequential computer whose processing speed is extremely low. Therefore, the neural network is now tried to be structured by exclusive electric circuits. There are various kinds of neural network structured by exclusive electric circuits, such as a multi-layered neural network. FIG. 8 shows the multi-layered neural network which has neurons having branch ability and integration ability and is provided with an input layer, an output layer and an intermediate layer of multiple layers interposed therebetween.
The multi-layered neural network shown in FIG. 8 has three layers of: an input layer composed of two neuron elements 111, 112 to which input vectors i=1, i=2 are respectively inputted, an output layer composed of two neuron elements 330, 340 respectively regarding outputs o=1, o=2, and an intermediate layer composed of six neuron elements 121 and an intermediate layer composed of six neuron elements 121-124, 131, 132 Formed in two layers. The intermediate layer is disposed between the input layer and the output layer. Between neuron elements in the respective layers, synapse groups 41-43 are disposed for setting coupling load therebetween. Each coupling load of synapses of the synapse groups 41-43 is changeable by learning.
A neural network agreeing to a recognition object is structured by leaning each coupling load of synapses of the synapse groups 41-43, changing each coupling load sequentially, adequately. As a learning method of each coupling load of synapses of the synapse groups 41-43, a back propagation method (BP method) is generally known in the art, which is much time consuming for learning and additional learning and whose learning algorithm is unsuitable for constructing the hardware.
A neural network circuit shown in FIG. 9 is proposed which is capable of high speed initial learning and additional learning and whose algorithm is suitable for constructing the hardware.
The neural network circuit in FIG. 9 is a neutral network developing the network structure in FIG. 8 into a tree-like branch structure, and is a three-layered neural network provided with an input layer, an intermediate layer and an output layer. The input layer is composed of neuron elements 11--11, 12-11 for only branch operation to which input vectors i=1, i=2 are respectively inputted. The intermediate layer is composed of 24 neuron elements 11-21-11-24, 11-31-11-38, 12-21-12-24, 12-31-12-38 for only branch operation which are formed in two layers, and has networks 13-1, 13-2 in tree-like branch structure in number of input vectors i=1, i=2 of the input layer (i.e., two). The output layer is composed of two output neuron elements (output units) 310, 320 for only integration operation, which respectively regard outputs o=1, o=2, and sums outputs from the upper 16 neuron elements 11-31-11-38, 12-31-12-38 of the intermediate layer. Between the intermediate layer and the output layer a synapse group 4 is disposed for setting respective coupling loads between the neuron elements. The coupling loads of each synapse of the synapse group 4 are changeable by learning. In FIG. 9, paths through 12-11-12-22-12-34-310 corresponds to paths through 112 - 122 - 132 - 330 in FIG. 8. Wherein, each coupling load of synapse between the neuron elements 11-11-11-38 and each coupling load of synapse between the neuron elements 12-11-12-38 are not learned and are set necessarily according to a value of the input vector inputted into the respective neuron elements 11-11, 12-11 of the input layer.
As an example of network system which depends on only the value of the input vector and sets necessarily, without learning, the coupling load of synapse in three-like branch structure, such as shown in FIG. 9, there is a network system called quantizer neuron which is disclosed in "Character Recognition System Using Network Comprising Layers By Function", Preliminary Material for Image Electronic Communication Society's National Conference 1990, pages 77-80 and "Multi-Functional Layered Network using Quantizer Neurons", Computer World '90, November. In this kind of network structure, the coupling loads of synapses of a final layer is changed independent form other synapses, which leads to high speed initial learning and additional learning and makes the learning algorithm suitable for constructing the hardware.
In the recognition method in the network system shown in FIG. 9, output values of the two neuron elements 310, 320 for only integration operation which are provided at an output layer are judged as to which is the largest and the address of the neuron element whose output value is the largest is made a recognition result. In the integration method in the neuron elements 310, 320 of the final layer for integrating outputs of the neuron elements 11-31-11-38, 12-31-12-38 of the intermediate layer, the respective output values of the intermediate layer and the respective coupling loads set in the synapse group 4 are multiplied and summed for integration.
The integration method in neuron elements are explained, with reference to FIG. 10.
In FIG. 10, the output neuron elements 310, 320 and the synapse group 4 are identical with those in FIG. 9. References f1 and f2 denote intermediate output values of the neuron elements 11-31, 11-32 in FIG. 9 respectively. In accordance with the above-mentioned references, the neuron elements 11-31-12-38 are branch points of an input signal, so that an output value from the neuron element 11-31 to the output neuron element 310 and an output value from the neuron element 11-31 to the output neuron element 320 are equal to each other and are indicated by f1. Respective coupling load calculations of synapses to the output neuron elements are executed by respective coupling calculation executing parts 4-11-4-22. The coupling load calculation executing part 4-11 is composed of a multiplier 112 and a coupling load coefficient W11 which is multiplied with the output value f1 corresponding to the intermediate layer to output a multiplied result. The coupling load calculation executing parts 4-12-4-22 have the same function as the coupling load calculation executing part 4-11, and have a different coupling load coefficient from one another. The integration calculations in the output neuron elements 310, 320 are expressed in following respective equations. The thus integrated output values of the neuron elements 310, 320 are judged as to which is the largest, and an address of the neuron element whose output value is the largest is made a recognition result.
output of output neuron element 310=W11.times.f1+W12.times.f2+ . . .
output of output neuron element 320=W21.times.f1+W22.times.f2+ . . .
The learning algorithm in network system shown in FIG. 9 uses the learning rule of Hebb, in which if the recognition result is false, the coupling load of the synapse group 4 to an output neuron element to be true is fortified until the value of the output neuron element to be true is the largest output value by a supervisor input in FIG. 9. As to the fortifying method, the coupling load coefficient is added according to the output value of the neuron elements 11-31-11-38, 12-31-12-38.
The fortifying method of coupling load of the synapse is explained, with reference to FIG. 11.
FIG.11 shows the coupling load W11 of FIG. 10 in enlarged scale. The intermediate layer output f1 is added to the present coupling load W11 according to a learning signal. The change in coupling load by learning is expressed as follows: EQU W11=W11+f1
In the multi-layered neural network structure which has the intermediate layer of tree-like branch structure, executes integration of synapses by output neuron elements of the final layer and executes learning by changing the coupling loads of the synapses of the final layer, the coupling load change is executed independent from the other synapses, which leads to high speed initial learning and additional learning and makes the learning algorithm suitable for constructing the hardware.
Recognition accuracy in the above multi-layered neural network of tree-like brunch structure is, however, low in case where a new unlearned data is recognized after an initial learning. The inventors have studied the reasons and consider the following is one of the reasons: in case where some kinds of input data are all identified by learning in the above neural network and variance of one kind of input data is small, a coupling load of the synapse group to the output neuron elements for recognizing a data similar to the input data is inflated at learning of the similar data so as to clarify a difference between the similar data and the one-kind input data, with a result that the similar data is misrecognized as an output result of output neuron element having as an input the inflated coupling load at next recognition of an unlearned data different from the similar data under such a condition since the inflated coupling load is extremely large compared with the other coupling loads of the unlearned data when the synapse group of the inflated coupling load is included in the synapse group to the output neuron element for the unlearned data recognition.
At the initial learning, the coupling load of synapse to the output neuron element is gradually increased according to the number of times at learning, which requires bit accuracy (bit word length) of coupling load and increases hardware in size which is required for coupling load memory. The above-mentioned references disclose that the coupling load memory of about 16 bits is required for 13-font learning of 62 kinds of character data according to a data group to be recognized. This means a large-sized hardware required for the coupling load memory.